cc @westonpace

I've checked locally that this fixes the TSan failures.

Submitted crossbow build: https://github.com/ursacomputing/crossbow/branches/all?query=build-226

Yes.

Hm, wait, I shouldn't unlock before notifying in MarkFinished().

Additionally, since we're handing around Executor* instead of shared_ptr or something, there's still the possibility that an I/O thread tries to use a dangling reference, right? The main thread has to make sure to block and wait for all I/O threads it spawns under all conditions.

Well, the idea is that the top-level task waits for all dependent tasks to finish, I think.

True, it's just a potentially easy pitfall (though it comes with the territory I suppose).

Things may get tricky if cancellation or early exit is involved indeed...

Seems Crossbow built Arrow master, not this branch. I threw up https://github.com/lidavidm/crossbow/actions/runs/748753207

The crossbow job passes!

Github Actions: https://github.com/pitrou/arrow/actions?query=branch%3AARROW-12379-tsan++
Travis-CI: https://travis-ci.com/github/pitrou/arrow/builds/223062391

Seemed the integration build timed out in Java but everything else passed.

Thanks @pitrou . A few thoughts (nothing worrisome):

Why deque instead of queue? I don't mind the change. I just want to make sure I'm not missing some piece of knowledge.

Things may get tricky if cancellation or early exit is involved indeed...\

Cancellation should still flush the task queue. Early exit (via bad Status) should still flush the task queue. It is important we do because in both cases the remaining tasks might need to do some cleanup to handle the error. Exceptions/assertions might not but we're already in crash territory there.

It's only for consistency, since deque is used for the threadpool executor.

https://issues.apache.org/jira/browse/ARROW-12379